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(54) Multiple description coding communication system 



(57) A multiple description coder 20 pairs two sam- 
ples and transforms the samples prior to quantizing and 
entropy encoding to improve coding efficiency while si- 
multaneously achieving satisfactory performance with 
regard to distortion. For the N-Sample case, the multiple 
description coder pairs the samples using two different 
algorithms, and then performs the pairing transforma- 
tion of the two variable case. One method assigns the 



lowest energy coefficients with the highest energy coef- 
ficients and so on until the middle coefficients are paired 
with each other. Another method assigns the highest en- 
ergy coefficient with coefficients having energy below 
some predetermined value. Image coders and decoders 
are also included. A method for classifying the coeffi- 
cients prior to performing the pairing transform is dis- 
closed, which classifies the coefficients according to the 
amount and location of their energy. 
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Description 

BACKGROUND OF THE INVENTION 

5 The present invention relates generally to communication systems, and more particularly to a communication 

system for communicating through unreliable channels, i.e., channels that have relatively high long burst errors and 
intermittent failures. 

An example of such a communication system, for example, is a wireless channel where long burst errors can occur 
due to the fading effect and where a physical link can be down temporarily because of an intermediate blocking. When 
10 transmitting image signals over such channels, these channel errors can lead to the loss of a large portion of the image 
if conventional image coders are used to code the image. For example, with a JPEG coder, the loss of a few bits in a 
block can make the current block and all the following blocks in a now undecodable. 

One way to enhance the reliability of communication system is by using multiple description coding (MDC) at the 
source coder. With MDC, several coded bit-streams (referred to as descriptions) of the same signal are generated. 
ts The coder is designed so that a better signal reproduction can be achieved with more descriptions, but the quality of 
the decoded signal is at least minimally acceptable even with only one description. 

FIG 1 illustrates a known two description coder and an associated channel model. It can be seen that the source 
end requires two encoders, which generate two descriptions; whereas the receiving end demands three decoders, 
among which only one functions at a time depending on whether stream one only, or stream two only, or both steams 
20 are received. ' 

Let R-, and R 2 represent the bit rates used for the two descriptions, respectively. Let E 2 (R 1 , R 2 ) represent the 
reconstruction error when both descriptions are received; and let E-, (R^ and E-, 2 (R 2 ) represent the reconstruction 
error when only first and second descriptions are available, respectively. The design of a multiple description coder 
can be stated as follows: Given R-,, R 2 , minimize E 2 (R-,, R 2 ) subject to a constraint 

25 ' 

E^ A (R^)<E^andE, 2 (n 2 )<E^ 0) 

where E e1 and E e2 are the maximum acceptable error if only the first and second descriptions are available, respectively. 
o0 In many applications, it is desirable to further require that the two descriptions by symmetric, i.e., 



35 



40 



so 



R ] = R 2 = R and 
E ]X (R) = E i2 (R) =E x (R)'and ~ (2) 

E e = £ = £ / ... 



.Next, we consider the design of such a symmetric multiple description coder. The problem can be stated as: Given 
R, minimize E 2 (R) subject to a constraint E^R)^ Ee. 

Let Pj, i = 0, 1, 2 represent the probabilities that none, one, or two descriptions' are received, respectively. If we 
assume that the probability that either description is lost is p, then the probabilities that 0', f, and 2 symbols are received 
are, , 



Px " = P(\~P) . ' (3) 

-p 2 = o-P) 2 



respectively. E(R, p 2 , p v p 0 ) can then be written as E (R, p). The average distortion achievable with the above symmetric 
two description coder is therefore; 
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E(R,p Qt p, ,p 2 ) - p 2 E 2 (R) + Pl E, (R) + p 0 E 0 (4) 

where E 0 stands for the distortion when none ot the descriptions is received. This error should be equal to the signal 
5 variance. Rather than putting a constraint on the distortion incurred when only one description is available (i.e. , requiring 
E, (R) < E e ), another plausible criterion lor designing a multiple description coder is: Given R and p. minimize the 
average distortion E (R, p). 

• Multiple description coding was first studied from a rate -distort ion theory point of view. The motivation behind MDC 
is to combat signal loss due to path failures. To. realize this goal, each coded stream (i.e., a description) must carry 
10 sufficient information about the original signal. Essentially this requires a certain degree of redundancy be embedded 
between the multiple descriptions. This will reduce coding efficiency compared to conventional signal description coding 
(SDC). 

Wolf, J. K., Wyner, A., and Ziv, J., "Source Coding for Multiple Descriptions," The Bell Svstem Technical Journal, 
vol. 59, pp. 1417-1426, Oct. 1980 showed that given a total bit rate 2R, the minimum distortion achievable by a single 
15 description coder, E min (2R), is less than the minimal distortion achievable by a multiple description coder when both 
descriptions are available, E 2 (R 1t R 2 ), if R-i t R2 = 2R - Wolf et al.. showed this using a rate. distortion analysis for an 
independent identically distributed binary source. 

. Ozarow, L, "On a Source Coding Problem With Two Channels and Three Receivers," The Bell Svstem Technical 
Journal , Vol. 59, p. 1921 , Dec. 1 980 also showed that the performance of a single description coder is better than the 
20 performance ot a multiple description coder when both descriptions are available in the case of an independent iden- 
tically distributed Gaussian source. 

• Specifically, Ozarow showed that if each coder is optimal in the rate distortion sense, i.e., E A j(R| ) is minimized tor 
a given Rj. lhen the joint coder will be far from optimal, i.e., E 2 (R^ R2) is much larger than the minimal distortion , 
achievable for this source E min (R 1 +R 2 ). The converse is also true: if the joint coder is optimal, i.e., E 2 (R-,, R 2 ) - E mjn 
25 (R 1 + R 2 ), then either one of the coders wilt be far from optimal, i.e., E^ j (R) is much larger than E min (R) for j = 1 or j 
= 2 or both. 

With real image and video signals, the redundancy in the signal (such as correlation among adjacent samples) 
can help reduce the loss in coding efficiency, but a certain amount of sacrifice in coding efficiency is unavoidable. 
However, this reduced coding efficiency is in exchange for increased robustness to long burst errors and/or channel 

30 failures. With SDC, one would have to spend many error-control bits and/or introduce additional latency to correct such 
channel errors. With MDC ( a long burst error or even the loss of one description does not have a catastrophic effect, 
as long as not all the descriptions are experiencing failures simultaneously. Thus, one could use fewer error control 
bits for each description. In this sense, the MDC is a way of performing joint source and channel coding. 

The first practical multiple description coder was proposed for speech coding. In this approach, the bitstream from 

35 a DPCM coder is split into even and odd sample packets. If an even (odd) sample packet is lost, data contained in the 
odd (even) sample packet are used to interpolate the missing even (odd) samples. It was shown that this coder works 
well beyond what the analysis predicted. This is in part because the analytical results hold true at the rate-distortion 
limit, while the bit rates in their proposed coder are nowhere close to the rate-distortion limit. In fact, there is sufficient 
redundancy left in the coder output, to permit subsampling and high quality interpolation. 

40 In another approach to MDC. multiple descriptions are obtained by quantizing each sample using two quantizers. 

The quantizers are designed so that the combination of the two quantizers leads to a finer quantizer, while each quan- 
tizer itself is a coarse quantizer. The simplest implementation of. this approach uses two quantizers whose decision 
regions shift by half of the quantizer interval with respect to each other. In this case, 2R bits are required to match the 
performance of a single quantizer with R+1 bits. Therefore, the loss" in coding efficiency* isquite significant, for R 

45 relatively high. Although more sophisticated quantizer mappings can be designed to improve the coding efficiency, a 
direct application of this approach to the original samples of a signal is not advisable in terms of loss in coding efficiency. 
Recently, this approach has been applied to transform coefficients in transform coders, with applications in both speech, 
image, and video. The loss in coding efficiency is less severe in this case, but still quite high, up to 30%. 

The present invention is therefore directed to the problem of developing a method and apparatus for performing 

50 multiple description coding that improves the coding efficiency while simultaneously providing similar estimation accu- 
racy. In addition, the present invention is directed to the problem of applying such a method and apparatus to image 
coding. 



SUMMARY OF THE INVENTION 



The present invention solves this problem for a two sample system by performing a pairing transformation on the 
two samples prior to quantizing and encoding. Each transformed sample is then quantized and encoded separately 
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and then transmitted over different channels. On the decoding side, the decoder first determines which channels are 
working, and entropy decodes and quantizes each channel. The decoder then performs an inverse pairing transform 
on the outputs of the quantizers. The inverse pairing transform performs three different transformations depending 
upon which channels are working, i.e., whether the first, second or both channels are working. 
5 According to the present invention for an N-Sample block, the block is first converted to N/2 sample pairs, which 

are then processed as in the two variable method.' On the decoder side, the process is similar to the two sample 
decoder, except that the pairmust be reordered to their original order after the inverse pairing transform. 

In one advantageous embodiment of the present invention for the N-Sample case, the block of coefficients is split 
" into two equal sized subblocks. Processing then proceeds as before. * 
' 10 in another advantageous embodiment of the present invention; the block of coefficients is segmented into two 

parts - the first part containing coefficients with relatively large energies and the second part containing coefficients 
with relatively small energies. Each of the parts is then processed differently. The first part coefficients are assigned 
to the two streams in the pairing arrangement discussed above, while the coefficients in the second part are alternately 
assigned to the two streams. 

15 According to another element of the present invention, the pairing of the coefficients occurs as follows. The coef- 

ficients with the largest energy are paired with the coefficients with the smallest energy in this part. Then the coefficients 
with the second largest and second smallest energies are paired, and so on until the coefficients in the middle are 
paired with each other. This pairing technique can be implemented in any of the two N-Sample MDC Coders discussed 
above. 

20 in an alternate embodiment of the pairing technique of the present invention, the coefficient with the largest energy 

can be paired with a coefficient having an energy less than a predetermined value, and so on until all pairing assign- 
ments are complete. One advantageous implementation of this ' embodiment occurs when the predetermined value 
equals the variance of largest coefficient times the estimation error divided by the difference between the variance of 
the largest coefficient and the estimation error. The end result of using this value is that each paired sample is coded 

25 with the maximum variance possible. 

According to yet another embodiment of the present invention, for certain types of applications the coefficients are 
placed in three groups -- those with the highest energies, those having energies in the middle and those with the lowest 
energies. The highest energy coefficients are duplicated on each channel, the middle level energy coefficients are 
paired as before, and the lowest energy coefficients are alternately assigned to each channel. On the decoder side, 

30 the process is similar to the other decoders, except that the duplicated samples must be separately processed from 
the other samples.. 

According to yet another embodiment of the present invention, the coefficients are classified according to their 
energy content. The different classes are then processed differently. On the decoder side the classification information 
that is received with the samples is used to determine the assignment of coefficients. 
35 ' it is particularly advantageous if the classifications of coefficients are defined as smooth, vertical, horizontal and 
random,' where smooth includes those coefficients having most of their energy in low frequencies or having a minimum 
amount of energy, where vertical includes those coefficients having most of their energy in the vertical direction, where 
' horizontal includes those coefficients having most of their energy in the horizontal direction, and where random includes 
those coefficients not having been included in any of the previous classifications. 



40 



BRIEF DESCRIPTION OF THE DRAWINGS 



FIG 1 depicts a prior art' multiple description" coder and decoder for the two description case: 
FIG 2 depicts a block diagram of the multiple description coder for the two variable case according to the present 
45 invention " ■ ■ " 

FIG 3 depicts the pairing transform Used in the present invention. 

FIG 4 depicts a block diagram of the decoder for the two variablecase according to the present invention for use 
with the coder of FIG 2. ~ 

FIG 5 depicts the inverse pairing transform used in the present invention. 
50 FIG 6 depicts a block diagram of the N-Sample Multiple Description Encoder according to the present invention. 

FIG 7 depicts one embodiment for pairing samples in the N-Sample case according to the present invention. 

FIG 8 depicts another embodiment for pairing samples in the N-Sartiple case according to the present invention. 

FIG 9 depictsa block diagram of the N-Sample Multiple Description Decoder according to the present invention 
for use with the Encoder of FIG 6. 
55 FIGs 10-11 depictsa block diagram of one embodiment an N-Sample Multiple Description Image Encoder accord- 

ing to the present invention. 

FIG 12 depicts an algorithm for classifying coefficients according to the present invention. 

FIGs 13-14 depict a block diagram for another embodiment of an N-Sample Multiple description Image Decoder 
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according to the present invention for use with the Encoder of FIGs 10-11 . 
DETAILED DESCRIPTION 

5 The present invention modifies a conventional block transform coder to realize a Multiple Description Coder (MDC). 

Given a block of L samples, as in the conventional transform coder, these samples are first transformed to L coefficients 
using a Discrete Cosine Transform (DCT), for example. It is possible to use another type of transform, however. 

Instead of directly quantizing and coding these coefficients, the present invention converts them to two streams 
of variables, with L72 variables in each stream. The variables in each stream are quantized, coded, and transmitted 
10 independently. Ideally; the conversion is such that if both streams are received, then the original transform coefficients 
can be recovered perfectly in the absence of the quantization error. If only one stream is received, the original transform 
coefficients can still be recovered to a certain accuracy. 

This is accomplished by segmenting the L coefficients into two parts. The first part contains N < L coefficients with 
relatively large energies, whereas the second part consists of L - N coefficients with small energies. These two parts 
T5 are then processed differently. 

As the first part contains significant coefficients, none of the. coefficients are allowed to be thrown away in either 
stream. On the other hand, repeating these coefficients in both streams will lead to too much loss in coding efficiency. 
To obtain a better compromise between the quality.and coding efficiency, a pairing mechanism is employed, which will 
be explained later 

20 : For the second part, the coefficients are split between the two streams evenly, i.e., even indexed coefficients are 
put into stream one, and odd indexed coefficients are put into stream two. If, for example, only stream one is received, 
then the odd coefficients are simply replaced by zeros. Because these coefficients have small variances, their loss 
doesnot cause a severe visual distortion. 

Returning to the first part, with the proposed pairing method, the N coefficients are grouped into N/2 pairs, and 
25 each pair is transformed into another two variables, which are put into the two streams separately. The pairing and 
• transformation are designed so that from any one variable, the original pair can be estimated to a satisfactory degree. 

This essentially requires that a large (in terms of variance). coefficient be paired with a small coefficient, so that the 
. large coefficient is almost coded twice. Obviously, doing so reduces the coding efficiency as with any multiple description 
» coder. A key issue is how to pair the coefficients so that a good trade -off -between the estimation accuracy and coding 
30 . efficiency is reached. , t 

Two Variable Case 

As described above, a basic constituent of the fylDC algorithm of the present invention is' the coding of pair of 
35 . variables. Instead of coding the two variables independently, they are transformed into two other variables, which are 
quantized and transmitted separately. In the receiver, if both transformed variables are available, they are inversely 
transformed to recover the original two variables. ,lf only one transformed variable is received, one must be able to 
estimate the original two variables. The transformation is such that from any one transformed variable, the original two 
variables can be estimated to a certain accuracy, and that the estimation errors for the two variables, are the same. 
40 Further, the transform is unitary so that the mean square quantization error added to the transformed variables is the 
same as that added to the original variables. t: ■ . . .... - . , ■ , . • , ■ / ; 

The Optimal Transform for Generating Two Equivalent Descriptions and the Optimal Linear Estimators 

45 in order to show what transform will satisfy our requirement, we first introduce Lemma 1 . We then state the optimal, 

transform and estimators in Theorem 1. The proof for both will be provided later. 

Lemma 1 : Given two RVs, A and B, assume they are uncor related,, with zero means and Variances o a 2 and o b 2 . If 

C= aA + pB * ' ' "' 1 ( 5 ) 

• then the best linear estimators for A and B from C are 

55 .. . A = y Q C and B = y t C \[ (6) 

with . • . . ♦ 
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10 

The mean square estimation errors are: 



2 



2 2 (1 2 2 
a o a + p o b 



u 2 2 2 
?fa~ 2 2 n 2 2 



ft 2 2 2 



15 e.a.min 2 2 rt 2 2 

a a + p a 



(7) 



(8) 



0) 



fa 



~e,b,min ' 



2 2 2 

a p 

2 2 n 2 2 

a o + p a. 



(10) 



Theorem 1: Given two RVs, A and B, with the same properties as specified in Lemma 1, it is desired to construct a 
linear transformation of (A : B) to (C,D}, so that the linear estimation errors of A and B from either C or D are the same 
25 and this error is minimized, and that the transformation is energy preserving so that variances o c 2 + o d 2 = a a 2 + a b 2 
The only solution is the unitary transform: 



30 



35 



40- 



AS 



C = ±{A*B) 

ft 

D = Ma-B) 

ft 



Further, the best linear predictors and the prediction errors are: 



co 7 2 > 



(") 



(12) 
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da 



10 



B(D) = y db D, with Y 



o: + o. 



(13) 



15 



"e,min 



F - R 

^e.s.min e.b.min 



2 2 
CO*, 



2 2 



(14) 



The Single and Multiple Description Coders for Two Variables 

20 Consider the coding and transmission of A and B through two independent and equal channels. Suppose we code 

and transmit these two RVs independently. If one channel undergoes a failure so that A or B is lost, because A and B 
. are independent, the best estimate of A or B is by the respective mean, and the corresponding estimation error is a a 
or o b 2 . if one RV, for.example A, has a significantly higher variance than B, then the failure of the channel carrying A 
will lead to a large error. . 

2S Theorem 1 suggests that we could transform A and B into C and D using'the unitary transform-described in (11): 

If both C and D are received, we can use an inverse- of (11), which is simply: 



30 



A , _L(C + D), 
ft 

B -. ±(C - D). 
ft 



(15) 



40 



45 



If only C or D are received, then we can use the estimator in (1 2) or (13) to estimate A and B. The estimation errors 
tor A and B from either C or D would be the same, as described in (14). - 

We will reler to the first coder which directly codes A and B as the two variable single descript.on coder (SDC2) 
and the second coder which codes C and D the two variable multiple description coder (MDC2). 

In the following subsections, we will analyze and compare the rate-distortion performances of these two coders. 
To simplify the analysis, we assume that A and B, hence C and D, are uniform random variables (RVs) and neither 
the SDC2 or MDC2, uniform scalar quantization is used lor quantizing A and B or C and D. In the followng analysis, 
we will make use of the well-known result in the quantization theory - ! when applying a uniform quantcer to a unrtorm 
RV the mean square quantization error E q is associated with the variance of the signal o* and the brt rate R by: 



ss 



(16) 



Note that we are mostly interested in the relative performance of the multiple description coding verses smgle 
description coding. Although the assumption that all the RVs are uniform is quite restrictive, the relat-ve gams/losses 
derived based on this asbumption should provide a good insight for the more general cases, as the same assumption 
is applied to both SDC and MDC. 

Analysis of the SDC2: 

Let R representee average number of bits specified for representing A and B. Thebest bit allocation that minimizes 
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the total quantization error for A and B is: 



10 



R t — iofiU- 



R b = R + ^logj 



/? - 1 log, A, 



R + -log, A. 
4 



(17) 



where 



75 



(18) 



20 



is the variance ratio between the two RVs. The quantization error is 



■2R 



(19) 



With SDC2, if both A (quantized A) and B (quantized B) are received, then the reconstruction error per symboi is 
simply the quantization error: 



30 



1 / *2 2 -2.R 

SDC22 = 2 (E <?.a + E q.b) = 4°a a b 2 



(20) 



If only A is received, and the missing B is estimated by its mean, then the reconstruction error is 

35 ( 

_ 2 
E q.a + °fa 



(21) 



Similarly, if only 5 is received, the reconstruction error is 



E q.b + <V 



(22) 



Assuming the probabilities of losing A and B are equal, the average reconstruction error per symbol is: 



AS 



^SDC2,\ 



l I 2 2 --IB 1,2 2\ 



(23) 



Finally, if both A and B are lost, the reconstruction error is: 

55 



1 2 2 
E SDC2,0 = 2(°a+°b) 
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Finally, the average error for given channel loss rates p 0 , p-,, p 2 is: 



"SCO P-fisDC2,7 + P\^SDC2A + P(fisDC2.0 



/ \ 

Px 



2 



Po 



(25) 



10 Analysis of the MDC2: 

For the MDC2, because 



15 



2 2.2 2 xo -2R 



it is natural to let R c = R d = R, and the associated quantization error is: 



(26) 



-2R 



(27) 



The reconstruction error in the presence of both C and D is simply the quantization error for C and D because the 
25 transformation in (11) is unitary. Thus, we have 



(28) 



30 |f, for example, only the quantized C,C = C + q c is received, the estimation error from C to A is: 



E ea (Q = E{(A-y ca Q 7 } = E{(A-y cii C) 2 } + E^) . (29) 

35 



40 2 2 4 

= E +- v L ~ + 2 nrn 

e,min * ca^q.c 2 1 2 2 * * 

45 

In deriving the above result, we have assumed that the quantization error is uncorrected with A or C. Similarly, 
we can obtain; 



so . . 

*e. 4 (0 " #7 * (3D 



° 0 + °b °o + :°b 
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2 2 4 



10 



15 



45 



= -^--rV- (32) 



7 7 4 
O O k Ol 

= -^+-^2- {33) 



The average reconstruction error per symbol when only C or 0 ' is available is: 



E = 1 

^MDCU y 



^K(Q + ^,(0) - |(£JZ5) ♦ j (34) 



2 2 4 4 

= -I^2 + 2 2 2 < 35 > 

Obviously, the error when both C and ft - and are absent is the same as in the SDC2, i.e., 

' 1 ' - • . 2 2 '■ ; . 

35 • ••• ., '• ' * , ,. E MDC2 1 0 ~ °a + °b ' ( 36 ) 

The average error for given, channel Joss rates p 0 , p 2 is: 

40 E MDC2 = P2 E MDC2£ + Pl E MDC2.1 + P() E MDC2,0 ( 37 ) 

Comparison of SDC2 and MDC2 



The inventors compared the various types of errors derived above for the SDC2 and MDC2. In the presence of 
both channels, the SDC2 performss better, while in the presence of one channel, the MDC2 performs better for relatively 
high bit rate. - ■• 

To obtain more quantitative comparison of these two coders, let us consider the quantization errors when the bit 
rate is fixed. From (2B) and (20), we find 



MDC ^- C * + ° e - 1+ ^>1 (38) 



55 which is equal to the ratio r between the arithmetic mean and geometric mean of a a 2 and a b 2 . This ratio is one only 
when o a 2 = c b 2 . Otherwise, it is greater than one, implying that the MDC2 will lose coding efficiency whenever the two 
variables have different variances, and the loss is more significant when they differ more substantially. 
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In terms of the loss in the Signal to Noise Ratio (SNR) when both channels are available, we have. 

ASNR 2 - SNR SDC2 ■ SNR MDC2 = 10log 10 ^ = 10 log 10 r W 

5 

It is interesting to note that the loss in the SNR is independent of the bit rate R. 

70 error. In this case, from (23 and 34-35), we have 



75 



,2 2 



2o„o 



'SDC2,\ 



J MDC2 m \ 



(40) 



20 



25 



30 



Therefore, 

E SDC2.1 _ 2 . (41) 
^002.7 

AS/vR, = 10log l0 |^ = 20log 10 r = 2ASWH 2 < 42 > 

1 fc MDC2.1 



This means the, 1dB loss in the coding efficiency wil, yie.d 2 dB gain in the estimation accuracy in the presence 

of one channel only. c innifirant when the two variables differ more in 

With MDC2, the loss in coding efficiency, i.e.. ASNR 2 is more ^^^^^ in the prese nce of one 

"EE bil ,a,es ^-d by « «o .*r ft. -~ «"« 



z^R 3 c _^o- 2R *-p ( 43 ) 

40 * * a 3 



we obtain the average bit rate required by the SDC2 is 

rn • v 

- ' ' 1 „V : 1 , ' ^°a°b : • • (44) 

RsDC = -2^ + ^ = 2 ,0g 2-E7 



With the MDC2, we can obtain in a similar way: 

«UDC2 = 2 < fl c + R d ) = "2 l09 2 ^ = "2 1092 ^ 
Therefore, the increase in the bit rate per sample for the same quantization error is: 



50 ' JS-lmn.i^ H5) 



55 
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Substituting (46) into (42), we obtain 

ASNR e = 40 lo 9l0 2Afl = 12Afl (47) 

ih* r 3 ," 8 ^ 6Xtra b " US6d by the MDC2 reduces the estimation error by 12 dB compared to the SDC2 On 
2£EK t' r R W v' A kn T R that 3n 6Xtra b " Can ° nly feduCe ,he ^"tizationerror in the SDC2 by dB " 

The N Variable Case 

In this section we consider the case where N RVsX n-i o m , u 
2 samples. The requirement is that if nni™ c ' '° be sp,lt lnto lwo s,reams ' each win W 

sen Jon of all N RVs ^ that ihe lonT , T^' ^ ^ Sti " ° b,aln 3 reasonab| y accura <e repre- 

wil, assume J^^^J^^TT TT °' eKher Channel is the same - As e* we 

wi t ho U tthe,ossSe R rS 



°l>< ' (48) 



^Z^Z^^^^T mS T qUanti29 ^ N WS ^ ° p,imal bil allocat - and 
be arbitrary, for ^^^^^ ^ TTa'^T ™ ^ C ° 6ffiCien,S ^ SPli " in9 
m = 1 , 2. . .N/2 into another *1 i r !? ? I 2m ' " ' " " ' N/2 lnto one s,ream ' and odd symbols, X 2nl+1 
uncorrected, the Zl^Z^^ ™«™ * T^' b6CaUSS the Symb °' S in ,he tw ° st -~ 
* equal to its variance. <^ k2f^rlSZl!E^ * Va ' U6 S ° ^ eS,imati ° n error is 

contain more significant coefficenS V ? f recons,ruction er ™ " one stream happens to 

N variable single de*=*J^£^ °' ** ,aad l ° 3 **« er ™ We wi " •*« to this coder as the 

stre^^ * «»« e the estimation .of the original N RVs from any one 

pair. For each pair A m f X ' B x I " T S^^USP 2 d ' SCUSSed h ' he previous section t0 each 
transmitted over twos^chann^^^ ^ ^ which are 

e,herC m orO mW ewi„ refer to ^^J^nJ^^^^^^^^ 
Analysis of the SDCN 



= log, 



2 

O 



. " = 1, 2, .... N 

(49) 



The associated quantization error for any symbol is: 
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^ = (n^2-» (50) 

The reconstruction error per symbol in the presence of both streams is simply the above quantization error, i.e., 



10 



if, 



(51) 



On the other hand, the reconstruction error when only stream one or two is available is: 



15 



20 



N 



[ \ 

SDCN.q,* + 52 "°n 
\n€ stream \ n e stream 2 J 



(52) 



25 



■*SDCN1.2 



Ye + T o 2 

n€ stream 2 ne stream 1 J 



(53) 



Assuming the probability that either stream is lost is equal then the average reconstruction error when only one.stream 
30 is available is: 



^,,4in^-^^|; 0 :^Eoi. : .. (54, 



35 



40 



The reconstruction error when both streams are lost is: 



i N 

- • ■= ±yv 



(55) 



The average error for channel loss rates p 0 , p-, and p 2 is: 

EsDCN " P£ E SDCN2 + Pi ^SDCNA + Pq E SDCN.O 



Pi + 



in of 



2 -w + 



\ 



-+p 0 



1 N 



(57) 



55 



BNSOCCID: <EP 0656955A1J_> 



13 



EP 0 856 956 A1 

Analysis of the MDCN 

In the MDCN, for each stream we are coding N/2 variables with variances 



2 

a - 



[°n{m,[) + °/i(m,2)) 



(58) 



10 



where a 2 n(m 1} and o 2 n(m 2) are the variances of A m and B m , respectively. The optimal bit allocation is 



15 



20 and the associated quantization error is 



(59) 



25 



'MDCN.q.m 



(60) 



The average reconstruction error in the presence of the two streams is thus. 



30 



F = F 



- (n 



(61) 



35 Extending the results obtained for the two variables case, the estimation error for A M tiom C m is 



MDC2.q.m 



4° Similarly, 



^e.b.c.m ^e.min + ^cb^i 



MDC2,q, m 



45 



SO 



e,a,d,/T) e.min^ * da 1 - MDC2,q,m 



^e,b,d,m ^e.min + Ydb^i 



MDC2,q.m 



The average reconstruction error per for the m-th pair in the presence of either stream is: 



(62) 

(63) 
(64) 
(65) 
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2E . + if 



I / 2 _ . 



)^MDC2,q,m 



(68) 



2°n(m,l)°n(m,2) 



14 4 \ 2 

°n(m,l) * a n(m,2)/ /-pr 2W, 

I 2 2 \2 Vll »/ • 

r^(m.l> + °n(/n.2)/ 



-2tf 



(67) 



75 



The average reconstruction error per symbol in the presence of one stream is: 



20 



,V/2 



i 2 Nn 2 



°/i(m,l) + °n(m,2) 



(69) 



The reconstruction error when both streams are lost is: 



25 



1 /V 2 
JV/1=I 



(70) 



30 The average error for channel loss rates p 0 , p t and p 2 is: 



E MDCN~ P2 E MDCN2 + P^MDCNA + Pq E MDCN,Q 



(71) 



35 Comparison of SDCN and MDCN 

Comparing (51 ) and (61 ), we get: 



40 



E 



MDCN. 2 



'SDCN;2 



A'/2 



T\r n 

t m - 1 



(72) 



45 



where 



50 



a n(m.1) * G n(m.2) 

fl 2 
^f/°n(m.1)^(m5) 



(73) 



is the flatness measure of the m-th pair, and r is the geometric mean of the flatness measures of the N/2 pairs. 

To compare the reconstruction error in the presence of one stream only, we again neglect the quantization error 
55 since it is small compared to the estimation error. Then 
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i N 



Lnrui #n On 2 a 2 
MDCN I \ ^ -°n{m,\)°n( m l) 



E 



15 



m 

, 52 \° n(m,\)° n(m,2 



n(m,2) 

20 " * = ~ (75) 

'.2) 



25 Mext, let us consider the bit rate increase incurred by the MDCN if we require that the average quantization error to 
be equal to a preset threshold, E q . This is a bit allocation problem where we want to minimize the total number of bits 
subject to a constraint on the total quantization error. The optimal solution for the SDCN is: 



2 

' and the average bit rate is ■ " ' 

35 -* . • • . : : ■••...'■< . 



With MDCN, the above result should be applied to each stream of N/2 samples, and the results are: 

2 

«i«w fm =l l °92T' rn = 1 ' 2 Nl2 (78) 



40 



45 



SO 
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R MDCN = j& R MD CN. m = T 1q &2 • (79) 



Therefore; the average bit rate increase is ■ 
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25 



M N = R UDCtl ~ Rsdc = -•og 2 (II f ) J W = - ,0 S/ (80) 



Pairing Schemes 

to We now consider what is the best pairing scheme so that the overall reconstruction error described in (71) is 

minimized for a given channel error ratep and a prescribed bit rate R. An analytical solution to this optimization problem 
is difficult to find. However, because there are only finite number of possible pairing options, N(N - 1)/2, it is possible 
to use an integer programming method to find the best combination so that the above error is minimized. 

Recall that in order to reduce the estimation error in the presence of one stream, one should pair symbols which 
75 differ significantly in variance. On the other hand, to increase the coding efficiency (reduce the quantization error for 
a given bit rate, or reduce the bit rate for a given quantization error), one should pair two symbols with similar variances. 
Minimization of the total reconstruction error in (71) is essentially a balancing act, weighted by the channel loss rate. 

For more reliable communications, one may require that the estimation error in the presence of one stream be 
less than a preset threshold, regardless of how small the channel loss rate is. In the following, we describe an algorithm 
20 which will partition N symbols into N/2 pairs so that the estimation error per symbol is less than or equal to a preset 
value, E e , and in the mean time, the loss in the coding efficiency is minimized. 

To see how this can be accomplished, we first introduce another Lemma. Lemma 2: 

Given one RV A, with variance o 2 for the estimation error to be less than or equal to E e (assuming E < a a 2 ), it 



must be paired with a RV 8 with variance 



30 

It can be shown that if o a 2 > 2 E e , then E e < a b 2 < 2E e . The reverse is also true. 

The above Lemma is easily derived by requiring the E e min in (4) to be < E e 
Pairing Algorithm A: Recall that, for a given a a 2 the smaller is c b 2 , the more loss in coding efficiency. would occur. 
Therefore the best o b 2 is the largest one that satisfies the constraint in (81). Based on this criterion, we derive the 
35 following pairing algorithm which, for each pair to be chosen, minimizes the loss in coding efficiency while satisfying 
the constraint on the estimation error. 

Step 1 Order the X n so that o, 2 > a 2 2 > ... a N 2 . Put them in a stack so theX 1 is at the top. Setm=1. 
Step 2 Select A m from the top element in the stack, and delete it from the list 
40 step 3. // the stack is not empty, choose B m from the remaining ones in the stack so that its variance is the largest 

one satisfying ■ ■ . i . i ... . 

■ o 2 h <-4^- ■ (82) 

a e 

If such an element is found, delete it from the list, and go to Step 4. Otherwise, if such an element does not exist 
or if the stack is already empty, set B m to zero. - 
so step 4 If the stack is not empty, set m = m + 1 , go to Step 2. 

Although the above algorithm can satisfy the constraint on the estimation error strictly, it does waste bits when 
pairing a symbol with a zero. Essentially, this symbol is coded twice and transmitted in separate channels. One may 
choose instead to use the bottom element in the current stack, if one can afford to relax the constraint on the estimation 
£5 error. 

Note that the above algorithm does not necessarily minimize the overall loss in coding efficiency over all N symbols. 
Rather it minimizes the loss in coding efficiency associated with each pair at a time, by taking the pair with the largest 
possible variance ratio. In this sense, it is a greedy algorithm. 
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Pairing Algorithm B: From (81 ), it is easy to see that if 



2 

! a.1 



2 

»a. 2 



(83) 



then 



fa.l . max 



'6.2. max 



(84) 
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This leads to the second, simpler, pairing algorithms Order X n so thatcfz o/> ... o^. Pair the k-th one with the (N- 
k)-th one in the list, i.e., A m = X m , B m = X N _ m+1 , form = 1 : 2 N/2. 

Note that this algorithm does not guarantee that the estimation error for each symbol is less than a preset value. 
But it avoids the overhead associated with the first algorithm where some symbols have to be paired with zeros. We 
have implemented both the Modified Pairing Algorithm A and the Pairing Algorithm B. For several test sets of variances, 
they have yielded the same pairing results. Therefore, in the actual coder simulation, we have only used the Pairing 
Algorithm B. 

Implementation of Multiple Description Coding in Transform Coders Proposed Multiple Description Transform 
Coding (MTC) 

Although the MDCN described in the previous section can be applied to any N uncorrected samples, it is helpful 
(in terms of reducing the reconstruction error when the transmission channel is not reliable) only if these samples have 
drastically different variances. As already hinted, the motivation for the proposed MDCN is lor increasing the reliability 
of a communication system utilizing a transform coder. The general structure of the proposed multiple description 
transform coder (MTC) is illustrated in FIG 6. ■ 

' In this coder, each block of L samples are transformed using a L-pt unitary transform. The transformed coefficients 
are then arranged in a decreasing order based on their variances. Then for a preset value of maximum estimation error 
E e , which should be larger than the largest quantization error allowed a q 2 , we find the. first N coefficients with o n 2 > E e . 
Then using a pairing algorithm, these N coefficients are split into N/2 pairs, A m , B ra> m = 1, 2, .... N/2. Each pair is 
transformed using (11) to yield C m , D m . All the C m are then put sequentially into stream one, while alt D m are put into 
stream two. For the remaining coefficients, even indexed ones are appended to stream one, and odd indexed coeffi- 
cients are appended to stream two. Each stream is then quantized and coded independently and transmitted through 
separate channels. " • ■■ ■ . \ 

If both streams are 'received, then the decoder performs the inverse transform in (.15) on each pair of received 
samples, w m to recover the quantized * coefficients^" , 6,- .If only one stream is received, say the one containing C m , 
then based on each C m , for 1 < n <N/2,A W " and 8 m are estimated using the linear estimator described in (13). For the 
remaining coefficients which are split by the even-odd rule, the missing coefficients are simply replaced with their 
means, which are assumed to be zeros; Finally, all the recovered coefficients are inversely transformed to produce the 
samples in the original domain. . , 

in the analysis presented in the previous sections, we have assumed each C m or D m is uniformly quantized subject 
to an optimal bit allocation within each stream, which makes the assumption that each quantized index is coded inde- 
pendently*. In practice, we can apply the ruh-length plus Huffman coding method of the JPEG coder to each stream, 
which has proven to provide significant* improvement over independent coding of individual coefficients. Our imple- 
mentation is based on the JPEG method which uses the L - 8 x 8 = 64 point two dimensional Discrete Cosine Transform 
(DCT) as the unitary transform. The DCT coefficients are first scaled using a perception-based normalization matrix 
(we used one recommended in the JPEG standard). It is-these scaled coefficients which are separated (by pairing and 
splitting) to two streams. The coefficients in each stream are uniformly quantized using the same quantization interval. 
The quantized coefficients are then coded using a combination of the run-length and Huffman coding, in exactly the 
same way as in the JPEG coder. The difference is that there are only U2 ~ 32 coefficients in each stream is treated 
as the DC component: 

A difficulty in applying the above coder to real images is that they are not statistically stationary The statistical 
properties vary significantly among different images, and among different blocks of the same image. Because the linear 
estimation coefficients need to be determined based on coefficient variances,' a mismatch in the actual variances and 
the assumed ones could lead to large estimation'error. We have evaluated two implementations. In the first one (to be 
referred to as MTC1), the variances of transform coefficients are obtained by averaging over all the blocks in an image. 
From these calculated variances, the coefficients are ordered and paired using Pairing Algorithm B. The linear esti- 
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mation coefficients are also determined based on these global statistics. With the second implementation (to be referred 
to as MTC2), the blocks in each image are classified into several classes so that the transform coefficients of the blocks 
in the same class have similar statistical properties. Then the coefficient variances of each class are calculated and 
ordered, so are the linear estimation coefficients. The two coders are discussed in more detail in the following. 

5 

The Basic Version: MTC1 

In.the basic version, MTC1, for each image, the coder goes through three passes. In the first pass, statistics of 
the coefficient means and variances are collected, and a coefficient scanning table is produced which orders the co- 
10 efficients based on their variance values. The estimator coefficients, 



T^U^W^W^^ 1 ' 2 '-^ (85) 

15 also determined and stored in a estimator table. Then in the second pass, the coefficients are grouped and quantized, . 
• and the statistics for the run-length symbols in each stream are collected. The Huffman tables for coding the DC and 
AC coefficients are designed based on the collected statistics from each stream. In the final third pass, the coefficients 
are converted into bit streams using the Huffman tables produced. 

20 The Enhanced Version with Block Classification: MTC2 

* The MTC1 assumes that all the blocks in an image have similar statistics. This is rarely true for real images. For, 
' example, blocks in a flat regions wilt have very different DCT coefficient distributions than blocks over a sharp edge in , 
a particular direction. Blocks covering edges in different directions will also have quite distinct characteristics. In order. 
25 to coop with this non-stationarity within an image itself, the second coder, MTC2, classifies the blocks, in each image 
into several classes so that the transform coefficients of the blocks in the same class have similar statistical properties. 

' In our present simulation, four classes are considered, which essentially correspond to blocks that are smooth, 
with horizontal edges, with vertical edges, and the rest. The classification is accomplished by examining the coefficient , 
- concentration pattern. If the energy of the DC coefficient exceeds a certain percentage (specified as a threshold) of . 
50 the total coefficient energy, then the block is classified as smooth; otherwise, if the energy along the first column (or - 
now) of the DCT block exceeds the preset threshold, the block is classified as with horizontal (or vertical) edges. A. 
threshold of 95% has yielded quite good classification results for several test images. , 

' ■ In order to evaluate the loss in coding efficiency of the proposed multiple description coders .against a good single, 
description coder, we also implemented a JPEG-like coder, in which all the DCT coefficients in a block are coded 
35 together using the run-length plus Huffman coding method. As with the MTC 1 andMTC2, the Huffman coders required 
in this coder are also optimized for the image being coded. In order to split the resulting bit stream into two symmetric 
' descriptions, we put the bits from even indexed blocks into stream one and those from odd indexed blocks into stream 
2. In the decoder, if only one stream (e.g. stream one containing even blocks) is received, the, blocks in the missing 
1 stream (e.g. the odd blocks) are simply reconstructed by assigning each pixel the mean image value. Obviously, the 
40 reconstructed image in the presence of only one stream would be very poor, with a severe checker-board artifact. We 
realize that some error concealment techniques can be applied to improve the reconstruction quality, which essentially 
interpolate the missing blocks from surrounding received blocks. :; ....... 

In both the STC and the two MTCs ( the DCT coefficients are first scaled by a visual based normalization matrix 
(we used the one recommended in the JPEG standard). These scaled coefficients or their transformed versions (in 
45 the MTC 1 and MTC2 case) are then quantized using the same quantization intervals. In all cases, the Huffman tables 
required in the coder are designed based on the statistics collected for the image being coded. For the STC, only one 
set of Huffman tables needs to be designed; for MTC-1, two sets of . tables are designed, one for each description; 
finally for MTC2, eight sets of tables are designed, one for each class and description. Because these tables are image 
dependent, they need to be coded as side information. In addition, with the MTC .1 , one set of the coefficient ordering 
50 vector and the estimation coefficients also need to be specified, whereas I MTC2, four sets.of such side information are. 
to be specified. In practice, one should be able to pre-design necessary Huffman tables as well as the coefficient 
ordering table and estimation coefficients (for the MTC 1 and MTC 2 case) based on a good collection of training images. 
The bit rate increase from that using image dependent tables should not be very significant, and should be similar for 
all the coders: In fact, we have evaluated the effect of using the same Huffman tables for two descriptions in the MTC 
55 ■ coders. The increase in bit rate is very insignificant. For the MTC2, we do add 2 bits per block which is required for 
. specifying the class index of the block. . , . 
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Summary and Discussion 
Proof of Lemma 1 and Theorem 1 

5 Proof of Lemma 1 . We first consider the estimation of A from C. Let A= y a C. We need to determine 7 a so that the 

estimation error 

E^ a =E{(A-A) 2 }=E{(A-y aC ) 2 } (86) 
is minimized. By the orthogonality principle,- we obtain 

E{(A-y a C)C} = 0 orE{AC}=y a E{<?} (87) 

Because E{AB} - 0, we get 
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2 

20 Y a = 2 2 2 2 (88) 

a o a + p o b 



ft 2 2 2 

25 ^0 t a,min 2 2 rt 2 2 

a o a + p g. 



Similarly; let B, we will find 
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(89) 



po* 

Yb= 2 2 n 2 2 . ( 9 °) 



a a a + p o 



, 2 2 2 2 2 rj 2 '2 

e,o,min a a a r o 



(91) 



Proof of Theorem 1. First let us consider the following problems: Find a and p so that if C- aA + PB is available for 
estimating A and B, the estimation errors lor A and Bare the same and this error is minimized. From Lemma 1 , this is 
easily accomplished by requiring 



This yields: 



2 2 

E =£-.=£ - = 



a„ + a 



» (93) 
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Now let us consider how to construct a pair of transforms to satisfy the requirement of Theorem 1 . From the above 
result, one transform can be obtained by letting = a, and the other transform can be constructed by letting (i = -a 
Any value of a will satisfy the equal estimation error criterion. However, to satisfy the energy preserving requirement, 
we must set 

5 ■ • . 

a = 4= ' < 94 > 

J2 

10 This leads to the transform given in Theorem 1 . 

The linear predictors and minimal prediction errors given in theorem 1 can be easily obtained from those given 
in Lemma 1 by setting 

15 a - -± (95) 
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p = ±4= (96) 

J2 



Two-Sample Multiple Description Encoder 



FIG 2 depicts a two-sample multiple description encoder 20 according to the present invention. The samples A 
25 and B are the resulting original samples that may have been transformed using a known Discrete Cosine Transform 
(not shown), but they need not have been first transformed. 

As discussed above, the present invention transforms the input variables and then separately quantizes and en- 
tropy encodes the transformed variables using known quantizers and entropy encoders. The resulting bits are then 
transmitted over separate channels. 
30 Referring to FIG 2, samples A and B enter the Pairing Transform 21 , which outputs samples C and D. The details 

of the pairing transform 21 will be discussed later with reference to FIG 3. The pairing transform outputs are then 
quantized by quantizers 22, 23, respectively, using the same quantization step size to produce samples C and p: , 
which are individually entropy coded using standard entropy coders 24, 25, respectively. The entropy-coded sample £ 
is sent on channel 1 while the entropy-coded sample is sent on channel 2. In an alternate embodiment, the quanti- 
35 zation in quantizers 22 and 23 can be offset as discussed earlier, if desired. 

The circuit diagram depicted in FIG 2 can be implemented in discrete logic or in software, as is known in the art. 
The quantizers 22, 23 and entropy encoders 24, 25 are well-known in the art, and need not be described further. 



Pairing Transform 



FIG 3 depicts the pairing transform used in the present invention. In the pairing transform 21, samples A and B 
are input to standard multipliers 32, 33, respectively, and are multiplied by the value stored in the memory storage 
location 31, which has been initialized at the value 1/72. To produce the output C, the outputs of the multipliers 32, 33 
are input to adders 34, 35, respectively, and are added. Similarly, to produce the output D, the output of the multiplier 
45 33 is subtracted from the output of the multiplier 34 in adder 35. Thus, the output of adder 34 is the transformed variable 
C, whereas the output of adder 35 is the transformed variable D. 

The circuit diagram depicted in FIG 3 can be implemented in discrete logic or in software, as is known in the art. 
The multipliers 32, 33 memory 31 . and adders 34, 35 are well-known in the art, and need not be described further. 

so Two-Sample Multiple Description Decoder 

FIG 4 depicts a Two-Sample Multiple Description Decoder 40 according to the present invention. The decoder 40 
is for use with the Two-Variable Multiple Description Encoder 20 depicted in FIG 2. 

First, the block "Determine which channels working" 41 determines whether channel 1, channel 2, or both are 
55 working by monitoring their inputs to the decoder 40. One possible embodiment for performing this function is to monitor 
the channel for the incoming error rate. For example, the device monitors the incoming bit stream to determine if the 
error rate is in excess of a predetermined value, tor example, to make the decision whether the input is working or not. 
If the error exceeds the predetermined threshold, then the device determines that the input is not working and sets the 



21 



BNSDOCID: <EP 06569SSA 1_l_> 




EP 0 856 956 A1 

switches as described below. If the error is within the predetermined threshold, then the device determines that the 
input is working and sets the switches as described below. Alternate embodiments for performing this function are well- 
known in the art, and could easily be implemented by one of skill in the art. 

Referring to FIG 4, the inputs from each channel are entropy decoded using standard entropy decoders 42, 43. 
5 The outputs of the entropy decoders 42, 43 are inverse quantized by standard inverse quantizers 44 : 45, respectively, 
to obtain samples^ andD. These samples C and& then enter the Inverse Pairing Transform (IPT) 46, which produces 
outputs A and fi . The Inverse Pairing Transform also receives as inputs the values of the three switches S1, S2, and 
S3, from the block "Determine which channels working" 41 . 

The circuit shown in FIG 4 can be implemented in discrete logic or software, as is well-known in the art. 
10 As shown in FIG 5, the Inverse Pairing Transform 46 contains eight standard memory locations 47-54, six standard 

switches 61-66, two standard adders 59, 60, and four standard multipliers 55-58. Switches S1, S2, and S3 are set 
according to the determination, as indicated in Table 1 . If both inputs are received ( C and the switch S 1 = 1 , S 2 = 
1 , and S 3 = b. On the other hand, if only one input is received, then the switch corresponding to that input is set to 
zero, i.e., if only C is received then = 0 and S 3 = a : and if only D: is received then S 2 = 0 and S 3 - c, 
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The eight memory locations 47-54 are initialized at the values shown in Table 
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Memory Location 


Value Stored in Memory Location 
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(continued) 



Mpmnru I ocation 


Value Stored in Memory Location 


7 




8 


•72 (104) ' 



30 



Depending on the position of switches St (either 0 or 1 ) (switch is shown as elements 61 -£2 for simplicity), the 
value C is multiplied by both the values in memory location 1 or 2 (elements 47-48) and memory location 3 or 4 
(elements 49-50). Similarly, depending on the position of switch S 2 (either 0'or 1) (switch S2 is shown as elements 
63-64 for simplicity), the value 0: is multiplied by both the values in memory location 5 or 6 (elements 51-52) and 
memory location 7 or 8 (elements 53-54). 

The outputs of multipliers 55 and 57 are added, as are the outputs of multipliers 56 and 58, to become the values A 
and ft, respectively, when switch S 3 (shown as elements .65-66 for simplicity) is in position "b." When switch S 3 is in 
position M a, u A is the output of multiplier 55 and b is the output of multiplier 56. When switch S 3 is in position B c," A is 
the output of multiplier 57 and fi is the outputof multiplier 58. 

The circuit depicted in FIG 5 can.be implemented in discrete logic or software, as is well known in the art. 

N-Sample Multiple Description Encoder 

RIG 6 shows the N-sample multiple description encoder 67 of the present invention. Essentially the N-Sample 
version of the present invention employs the Two-Sample Multiple Description Encoder 20 as building blocks. Three 
additional blocks are necessary. First, the decide and assign pairing block 68 converts the incoming samples to paired 
samples, which can then be processed by N/2 Two-Sample MDCs 20, each of which consists of a pairing transform 
(69, 70, or 71 ) and two quantizers (76, 77 or 78 : 79 or 80, 81 , respectively). 
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where q m is the quantizer step size to quantize pair m. Thus, to maintain the error the same, each quantizer may have 
a potentially different quantizing size. 

Second, the Assign Channel and Multiplex for Channel block 73 receives the outputs from all of the MDCs and 
assigns channels and multiplexes them into a format suitable for the channels. Finally, the bit size and step size as- 
signment 72 controls the quantization level of the quantizers. Note that 

Returning to the input, the N samples X 1 through X N are input to'the block "Decide and Assign Pairing- 68, which 
outputs the samples by pairs as described below. For convenience, we rename the paired samples as A 1 and B v A 2 
and B 2 , .... A^ and B N/2 . The N/2 pairs are input to the N/2 Pairing Transforms 69-71 to output C, and D 1t C 2 and 
D2 and C N/2 and D NA >. Each of the Pairing Transforms is as depicted in FIG 3 and described above. 

Each member of the pair C; and D, are quantized by quantizer Q; (76-81 ) with a step size determined from the "Bit 
Allocation and Quantizer Step size Assignment" block 72. In "Assign channel and Multiplex for Channel" 73 samples 
C lf C 2 , ... and C N/2 are assigned and multiplexed for channel 1 while samples D v D 2 , ... and D^ are assigned and 
multiplexed for channel 2. These multiplexed samples are then entropy coded using a standard, entropy coder 74, 75 
typical in image compression systems. The outputs of the entropy coders 74, 75 are then sent on their respective 
channels. 

Determine Pairing Assignment A 

The algorithm for determining the pairing assignment A according to the present invention is depicted in FIG 7. 
First, the samples X N are ordered according to their variance o 2 . such that o, 2 > o 2 2 > ••• > c N 2 The list is formed with 
the sample (X,) with the largest variance at the top. The maximum tolerable estimation error E e is preset, which is a 
value that can be selected by one of skill in the art. 

The incremental counter is set to one, and the first sample is selected from the top ol the list and set equal to A m . 
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The first sample is then deleted from the list. Its variance is set to c a 2 . Next, the stack is examined to determine if all 
of the samples have been assigned. If so, i.e., YES, then B m is set to zero, and flow returns to "Is stack empty" 90. If 
the stack is not empty, i.e., NO from element 86, then the list is examined to determine if there is a element in the list 
with a variance such that: 

2 - a * fc * (106) 



a < 

b 



If YES, then that element is assigned to be B m and is then deleted from the list. If the answer from element 87 is NO, 
then B m is set to zero, and flow returns to "Is stack empty" 90. The stack is then examined to determine if it is empty 
(element 90), and if the answer is YES, flow ends. If the answer is NO, the counter, is incremented and flow returns to 
element 85. * 

Determine Pairing Assignment B 

FIG 8 depicts the pairing assignment B used in the present invention. First, the samples are ordered as in pairing 
; assignment A, with the largest variances first. The incremental counter is set to one. The variables are then paired 
20 largest with smallest, moving towards the middle so that the middle two variables are paired together. First, the sample 
with the largest variance (X-,) is set to A-,, while the sample with the smallest variance (X N )Js set to B 1 and the two 
variables ^ and B 1 are paired. Similarly, the next samples with the largest (X 2 ) and smallest (X N _-,) variances,' respec- 
tively, are paired. The process continues until the middle two samples are paired (element 98), e.g., X N/2 and X N/2 +v 

25 N-Sample Multiple Description Decoder 

FIG 9 depicts the N-Sample Multiple Description Decoder 100 according to the present invention. Similar to the 
N-Sample Encoder 67, which used the Two-Sample Encoder 20 as building blocks, the N-Sample Decoder 100 uses 
the Two-Sample Decoder 40 as building blocks. The Inverse Pairing Transforms 110-112 are as described in the Two- 
30 1 Sample Decoder 40 (FIG 4), as are the inverse quantizers 104-109, the "Determine which channels working 1 * block 
114, and the entropy decoders 101-102. The only additional blocks are the I'Demultiplex pairs from channel" block 103 
and the "Reorder pairs as samples" block 113., 

As in the 2-sample multiple description decoder 40 (FIG 4), the block "Determine which channels worKing" 100 
determines whether channel 1 , channel 2, or both are working by monitoring their inputs to the decoder 1 00,. Switches 
35 S1 , S2, and S3 are set as before, in Table 1 . The inputs from the channel are entropy decoded, using standard entropy 
decoders 101, 102. The outputs of the entropy decoders 101, 102 are demultiplexed into pairs in "Demultiplex pairs 
* '■ from channel" 103*. Each member of each pair is inverse quantized using standard inverse quantizers 104-109. The 
pairs then enter their Inverse Pairing Transforms^ 0-1 12, which operate as described in FIG 5, using the switch po- 
sitions generated from the "Determine which channels working" block 114. The outputs of the Inverse Pairing Trans- 
40 forms 110-112 are finally, reordered as were the original samples in block 113. . 

Multiple Description Image Encoder 

Next, we discuss application of the present invention to the field of image coding. Image coding presents unique 
45 problems for encoding algorithms based on statistics. Since all blocks in an image "do not have the same statistical 
characteristics, it is useful to apply the same algorithm using different parameters to different parts of the image. For 
example, it is useful to classify different blocks in the jmage as having certain statisical characteristics and then to 
process each block using parameters that are tailored to its class of blocks. Since the decoding of each block then 
depends heavily on knowing the correct parameters to use for each class of blocks, that information should be sent 
so on both channels. Furthermore, recognizing that sometimes simply pairing coefficients using the pairing transform may 
not provide sufficient redundancy, it may also be useful to duplicate the most important coefficients in both channels. 

The multiple description image encoder works with two passes. As shown in FIG 10, the image is input into the 
"Decompose into blocks" element 251, and passed through a Discrete Cosine Transform 252, which outputs the L 
DCT coefficients to the block classification 253. The operation of the block classification will be described with reference 
55 to FIG 12 below. ' ' 1 " ' - 

The block classification 253 outputs the L DCT coefficients and the classifications to the "Compute variance of 
DCT coefficients in class" 254, as well as to the "Assign Coefficients to be Duplicated, Paired or Alternated" element 
258 in FIG 11. 
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Based on the desired redundancy and the L variances output by the "Compute variances for each class" element 
254, the "Determine Number of Coefficients to Pair (N) and Duplicate (D) for each class" element 255 outputs the 
values N and D, which represent the number of elements to pair and duplicate, respectively. 

The "Code Variances and N and D for each class" element 257 then outputs the classification and the values N 
s and D to both channels, as shown in FIG 10. 

The "Determine Pairing assignment for each class" element 256 takes for each class the L variances and the value 
N and pairs N of the coefficients X 1 to X L . The remaining are either duplicated or alternated. The number of alternated 
coefficients is A = L- D- N. The pairing assignment is then sent to the "Assign Coefficients ..." element 258 in FIG 11. 

The "Assign Coefficients to be Duplicated, Paired or Alternated" element 258 identifies the coefficients to be du- 
10 plicated and outputs these (Xj . .. X D ) for transmission on both channels. Thus, there are D coefficients to be duplicated. 
These coefficients represent those coefficients that carry the most information (i.e., the highest energy) and cannot be 
lost over either channel. > ■ 

The "Assign Coefficients" element 258 also identifies the coefficients to be paired, and outputs these as paired 
samples (A, , B 1 , ... A N/2 , B N/2 ) to be transmitted on different channels. Thus, there are N samples that are transmitted 
is over two channels, hence N/2 samples are transmitted on each channel. These coefficients represent the coefficients 
that need not be sent over both channels, but cannot be sent in an alternating manner without sjgnifrcantjoss in signal 
quality or information. 

Finally, the "Assign Coefficients" element 258 outputs the coefficients to-be alternated, which consists of the re- 
maining coefficients, i.e., A = L - D - N coefficients remain to be transmitted. As these coefficients represent the jeast 
20 significant coefficients, i.e., they have the lowest energy, these coefficients can be transmitted in an alternating manner. 
If any coefficients get lost, they can be replaced with zeros, without significant loss in signal quality or information. 

The paired coefficients A v B 1 ... A N/2 , B N/2 are passed through pairing transforms 259-261 as described with - 
reference to FIG 3. ' " • ■ 

The outputs from the "Assign Coefficients" 258 are then quantized in standard quantizers 262-264 and 271-276 
25 and the outputs from the pairing transforms 259-261 are quantized in standard quantizers 2660-270. The outputs from : 
the quantizers 262-276 are then passed to the "Assign and Muxx, -X D to both channels, Assign and MuxC, -C H/1 to 
Channel 1,and£>, to Channel 2, and Assign and Mux X D .- N :, tochanneM and& D \ N + 2 -X t to channel 2" 277. 
The multiplexer 277 duplicates the X, -X 0 ' and outputs X t -X D to each "entropy encoder 278, 279. The multiplexer:277 
also outputs the paired samplesC, -C h . to entropy encoder 1 (278), andD, -0 N to entropy encoder 2 (279), so that . 
30 these are sent over different channels. Finally, the multiplexer 277 sends the samples V N . . A with odd indices to 
entropy encoder 1 (278), and the samples Vs.. -5? L with even indices to entropy encoder (279). 

The entropy encoders 278, 279 are standard entropy encoders, which output the coded samples to channels 1 
and 2, respectively. The classification information is sent on both entropy encoders 278, 279. 

35 Block Classification 1 - > : ' v 

FIG 12 depicts the block classification algorithm used in the present invention. First, the values E, L„H, and, V are 
"initialized to zero, and the threshold T n is initialized to 10. This could have. been: a different value, and is used for 
illustration purposes only. E represents the overall energy of the block/ L . represents the low energy of the block. H 
40 represents the horizontal energy of the block. V represents the vertical energy of the.block. . ; - . 

Then, the overall energy in the block, E, is computed as shown in element 143 and as in (107). 

* i ■ ' 1 

Next, a measure of the 'low-frequency energy in the block, L, is computed as given in element 144 and as in-(108). 

' - l --EE 4 : - ' > < 1Q8) ' 

. ' = i/ =i . ... i . . . 

55 Then, a measure of the horizontal energy, H, and the vertical energy V are computed as in element 145 and (109 

and 110).- * ■ ...:.*. 
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In the next step 146, threshold T 2 is set to be 95% of the overall energy in the block: T 2 = 0.95 *E. Then : it is tested 
whether L > T 2 or E < T v If YES, the block is assigned to the class "SMOOTH." If NO, the block is further tested whether 
V > T 2 , If YES, the block is assigned to the class "VERTICAL," IfNO, the block is again further tested whether H > T 2 . 
If YES, the block is assigned to the class "HORIZONTAL." If NO, the block is assigned to the class "RANDOM." 

Those skilled in the art would be able to use other algorithms to classify the blocks. Also, they could use a different 
equations to compute measures of the low-frequency energy, the horizontal energy, and the vertical energy. This al- 
gorithm and these equations are given merely for illustration purposes. 

Multiple description image decoder 

The multiple description image decoder works similarly to the , multiple description decoder of FIG 9. First, the 
channel inputs are input to the "Extract variances and N, D for each class" blocks 282, 283, which parses the input 
bitstream to determine the location of the variances and values of N, D and sends N and the variances to the "Determine 
Pairing Assignment for each class" block 284. The "Extract variances and N for each class" block 282, 283 also outputs 
the remaining bitstream parts containing the block classification and DCT coefficients, which is input to the respective 
entropy decoders 285, 286. The value D is sent from the "Extract variances" 282, 283 elements to the "Demux dupli- 
cations and pairs from channels and order alternating coefficients" element 287. 

Also, the channel inputs are sent to the "Determine which channels working" block 281, which operates as de- 
scribed previously, and outputs the values of the three switches, S1, S2, and S3 to be used by the Inverse Pairing 
Transforms 298-300. The "Determine which channels working" element 281 also outputs the value S 3 tothe "Demux ... 
" 287 for use in identifying the duplicated coefficients. 

The "Determine pairing assignment for each class" block 284 outputs the pairing assignment and passes on the 
values of N and the variances to the block 288 that computes the predictors for each pair and each class using the 
equations (12) and (13). The predictors are stored in their respective memory locations in the respective Inverse Pairing 
Transforms 301-303. 

The output from the entropy decoding and the block classifications and DCT coefficients are sent to the "Demux 
duplications and pairs from channel and order alternated coefficients" block 287, which also receives as input the 
pairing assignment computedpreviously The "Demux ..." outputs the duplicated coefficients, which are inverse quan- 
tized in inverse quantizers 289-291 . If S3 = a the duplicated coefficients are taken from channel 1 . If S 3 - c the duplicated 
coefficients are taken from channel 2. and If S 3 = ,bm then they can be taken from either channel. The "Demux.. ." block 
287 outputs the paired coefficients which are inverse quantized using quantizers 291 -297 and input to their respective 
Inverse Pairing Transform 301 -303, respectively. The VDemux..." block 287 also outputs the ordered coefficients that 
were alternated, which are then inverse quantized using quantizers 296-300. ; 

Referring now to FIG 14, the output of the Inverse Pairing Transforms 301 -303 are then sent to the "Reorder pairs 
as samples" block.304, which reorders the pairs in their original coefficient order. Then, the inverse DCT 305 is applied 
to all the DCT coefficients for the block. Each image-block is sent to the "Reassemble image from blocks" block 306 
which outputs the decoded image, • * 

Claims 

1 . A method for coding two samples comprising: 
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a) transforming the two samples into two coefficients having substantially equal energy; 

b) quantizing each of the two coefficients separately; and 

c) entropy coding each of the quantized coefficients separately. 

s 2. The method according to claim 1 . wherein the transforming step further comprises transforming a first sample (A) 
and a second sample (B) into a first coefficient (C) and a second coefficient (D) according to the folbwing equations: 

C= ~ (A + B), 



15 



20 



25 



30 



35 



3. The method according to claim 1 , wherein the transforming step further comprises transforming the two samples 
into two coefficients using a unitary transform. ' * 

4. A method for performing multiple description coding on a block of coefficients, comprising the steps of: 

* a) pairing coefficients within the block of coefficients to form a plurality of coefficient pairs; 

b) transforming each coefficient pair into a first transformed coefficient and a second transformed coefficient; 

c) quantizing each of the first and second transformed coefficients separately; and 

d) assigning the first and second quantized and transformed coefficients to different channels. 

5. The method according to claim 4, further'comprising the step of: ' - 

f) entropy encoding the first and second quantized and transformed coefficients separately prior to transmis- 
sion. ^ . ... 

6. The method according to claim 4, wherein the step of transforming further comprises transforming each coefficient 
pair (A, B) into a first transformed coefficient (C) and a second transformed coefficient (D) according to the following 
equation: , ' ' ■ 

. . ... . C= i ( " +B) ; ..." 



40 D = ±(A-B). 



7. The method according to claim 4, wherein the transforming step further comprises transforming each coefficient 
pair into the first "and 'second transformed coefficients using 5 a unitary transform' such that- -the first and second 

45 transformed coefficients have substantially equivalent energy; - " 

8. The method according to claim 4, wherein the step of pairing further-comprises pairing samples with relatively 
large variances with samples with' relatively small variances. -" ' v * ! - • ...... 

50 9. : The method according to claim 4, wherein the step of pairing further comprises: 

(i) ordering the coefficients according to variance from highest to lowest; 

(ii) successively pairing each coefficient (A) from a top of the list with a first unpaired coefficient (B) moving 
down the list whose variance (a b 2 ) satisfies the following equation: 

55 
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2 < °a E e 

5 ' 

where a a 2 is the variance of the coefficient (A) to be paired. 

10. The method according to claim 4/wherein the step of pairing further comprises: 

to ' (j) ordering the coefficients according to variance from highest to lowest; 

(ii) successively assigning a coefficient (A) with the largest variance with a sample (B) from the bottom of the 
list; and 

(iii) deleting the assigned coefficients from the list and repeating steps (i) and (ii) until no coefficients remain 
on the list. 

15 ' . 

1 1 . The method according to claim 4, wherein the step of quantizing comprises quantizing each transformed coefficient 
using a quantization interval that is offset from the other. . , ' ...... 



12. The method according to claim 4, further comprising the step of controlling the bit size and quantizer step size for 
20 each quantizer separately. 

13. The method according to claim 12, wherein the step of controlling further comprises maintaining the bit size and 
the quantizer step size the same for the two quantizers assigned to each coefficient pair. 

25 14. A method for decoding a signal received over two channels using multiple description coding, comprising the steps 
of: 

a) determining which of the two channels are working; 

b) entropy decoding each output from the two channels separately; 

30 c) inverse quantizing the output from each of the entropy decoders separately; and 

d> inverse pairing and transforming the outputs from the inverse quantizers using information regarding which 
channels are working.. 

15. The method according to claim 14, wherein the step of inverse pairing further comprises transforming the outputs 
3$ (C, D) from the inverse quantizers into two samples according to the following equation: 

40 

45 16. The method according to claim 14, wherein the inverse pairing step further comprises transforming the outputs 
(C, D) from the inverse quantizers into two samples using a unitary transform. 

17. A method for coding an image comprising the steps of: 

50 a) decomposing the image into blocks; 

b) transforming samples in each block into a block of coefficients; 

c) classifying each coefficient block into at least two classes; 

d) calculating the variance of the coefficients in the class; 

e) determining the number of coefficients to pair and assigning a first plurality of coefficients to be transmitted 
55 over both channels; 

f) determining the number of coefficients to duplicate and assigning a second plurality of coefficients to be 
paired and transformed; 

g) determining the number of coefficients to alternate and assigning a third plurality of coefficients to be sent 
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over the first channel and a fourth plurality of coefficients to be sent over the second channel; 

h) pairing the second plurality of coefficients to form a first plurality of paired coefficients; 

i) transforming the first plurality of paired coefficients into a second plurality of paired coefficients; 
j) quantizing each coefficient in the first, third, and fourth plurality of coefficients separately; 

k) quantizing each coefficient in the second plurality of paired coefficients 1 separately; 

I) outputting a quantized version, of the first plurality of coefficients to both channels; 

m) outputting a quantized version of the third plurality of coefficients to the first channel; 

n) outputting a quantized version of the fourth plurality of coefficients to the second channel; and 

0) outputting a quantized version of one coefficient of each pair of coefficients within the second plurality of 
coefficients to the first channel and 1 outputting a quantized version of the other coefficient of each pair of. 
coefficients within the second plurality of coefficients to the second channel.. 

1 8. The method according to claim 1 7, further comprising the step of entropy coding each channel separately. 

19. The method according to claim 17, further comprising sending the classification information in each channel. 

20. The method according to claim 1 7, wherein the step of classifying comprises: 

a) determining the total energy within each block; 

b) determining the low frequency energy within each block; 

c) determining the horizontal energy within each block; 

d) determining the vertical energy within each block; . 

e) classifying a block as smooth if most of its energy is low frequency energy or its total energy lies below a. 
predetermined threshold; 

' ' f ) classifying a block as vertical if most of its energy lies in vertical coefficients; 

g) classifying a block as horizontal if most of its energy lies in horizontal coefficients; and 

h) classifying a block as random if the block fails to meet the other classifications. 

21 . A method for decoding a coded image sent over two channels comprising: 

* : a) determining which of the two channels are working and controlling a plurality of inverse pairing transformers 
using this information; 

b) extracting a plurality of variances and a number (N) of paired and duplicated (D) coefficients from each 
channel; - - • - •......*.'•:>- 

c) determining a pairing assignment for each classification; : ; . 

d) entropy decoding each received channel separately; 

e) computing predictors for each pair and class; 

f) demultiplexing the duplicated, paired and alternated coefficients using the number of paired (N) and dupli- 
cated (D) coefficients, information regarding which channels are working, the paring assignment and the clas- 
sifications; 

g) inverse quantizing all of the demultiplexed coefficients; 

h) inverse pairing and transforming the quantized versions of the paired coefficients and reordering the output; 

1) inverse transforming the coefficients; and 
j) reassembling the image from the blocks. 
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